Ephemeral Document Clustering for Web Applications

We revisit document clustering in the context of the Web. Speci cally, we investigate on-line ephemeral clustering, whereby the input document set is generated dynamically, typically by search results, and the output clustering hierarchy has a short life span, and is used for interactive browsing purposes. Ephemeral clustering for interactive use introduces several new challenges. It requires an e cient algorithm, since clustering is performed on-line. It also requires high precision, because users who are not domain experts are less tolerant to errors, and because the resulting hierarchy is fully automatically generated, as opposed to o -line clustering in which the hierarchy is often manually modi ed. Finally, interactive clustering requires a presentation layer that enables users to e ectively browse the hierarchy, including visualization techniques and automatic annotations of the hierarchy. We present new concepts, techniques and algorithms that tailor clustering to these requirements. More speci cally, we improve precision by feeding the clustering algorithm with more precise pro les using \lexical a nities" as indexing units. We give an optimal complete-link Hierarchical Agglomerative Clustering algorithm, with O(n) time complexity. Finally, we provide a visual yet light Java-based implementation of the presentation layer. We demonstrate the utility of ephemeral clustering through a search assistance utility that embodies these ideas. keywords: information retrieval, hierarchical clustering, document clustering IBM Research Report RJ 10186, April 2000. IBM Research in Haifa, MATAM, Haifa 31905, ISRAEL. Email: [email protected] IBM Almaden Research Center, 650 Harry Road, San Jose, California, USA. Email: [email protected] Department of Electrical Engineering, Technion, Israel Institute of Technology, Haifa 32000, ISRAEL. Email: [email protected] Department of Computer Science, Carnegie-Mellon University, Pittsburgh, PA, USA. Email: [email protected]